User-Level Network Interface Protocols

53 M odern high-speed local area networks offer great potential for communication-intensive applications, but their performance is limited by the use of traditional communication protocols, such as TCP/IP. In most cases, these protocols require that all network access be through the operating system, which adds significant overhead to both the transmission path (typically a system call and data copy) and the receive path (typically an interrupt, a system call, and a data copy). To address this performance problem, several user-level communication architectures have been developed that remove the operating system from the critical communication path. 1,2 In this article, we describe six important issues to consider in designing communication protocols for user-level architectures. The issues we have selected focus on the performance and semantics of a communication system. The first, the data transfer mechanism a protocol uses, significantly affects latency and throughput. Direct memory access usually gives the best throughput in user-level communication, but to avoid unnecessary copying, sophisticated address translation mechanisms are needed, another important design issue. To make user-level protocols feasible in a multiprogramming environment, designers also need protection mechanisms that do not rely on operating system intervention. Network packets are received using some control transfer mechanism—either polling or interrupts, each of which has its advantages and drawbacks. The last issues we focus on are reliability and multicast, which in modern networks can be supported both on the host processor and the network interface. To provide a basis for analyzing these issues, we present a simple network interface protocol for Myricom's Myrinet network. 3 As the sidebar " The Myrinet Network " describes, Myrinet has a programmable network interface. Researchers can thus explore many protocol design options, and several groups have designed communication systems for Myrinet. In this article, we refer to 11 such systems. Table 1 lists the characteristics of each. As the table shows, the systems differ significantly in how they resolve the design issues we describe, but all aim for high performance and provide a lean, low-level, and more or less generic communication facility. Figure 1 shows the operation of our simple network interface protocol for Myrinet. Operation begins when a user process of the sending host—host A—invokes a simple send primitive to send a data packet. Because the protocol sends packets with a maximum payload of 256 bytes, users must fragment their data so that each fragment fits in a packet. Send performs two …